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Abstract 


The Study of Deeper Learning: Opportunities and Outcomes-funded by the William and Flora Hewlett Foundation-aimed to determine 
whether students attending high schools with a mature and at least moderately well implemented approach to promoting deeper 
learning actually experienced greater deeper learning opportunities and outcomes than they would have had they not attended these 
schools. In this report-our second in a series of three-we focus specifically on the key question: Did students who attended deeper 
learning network schools have more opportunities to engage in deeper learning than would likely have been the case had they not 
attended the network schools? This question addresses a fundamental assumption that underpins the deeper learning initiative: 
that a well implemented approach to deeper learning can result in greater opportunities for students to develop deeper learning 
competencies. This analysis includes 11 pairs of matched deeper learning network and comparison schools in California and 
New York. While we collected a wide range of data for this study, we relied primarily on student survey data and examples of the 
work teachers assigned to students (teacher assignments) for this analysis. 

Key takeaways include the following: 

1. On average, students who attended the network schools in the study reported greater opportunities to engage in deeper 
learning than did similar students who attended non-network schools. Positive effects were found across all measures including 
opportunities for complex problem solving, opportunities to collaborate, opportunities to communicate, opportunities to 
learn how to learn, opportunities for creative thinking, opportunities to receive feedback, opportunities for assessments aligned 
with deeper learning, opportunities for interdisciplinary learning, and opportunities for real world connections. 

2. Since the study schools served substantial populations of students living in poverty and, in some cases, large populations 
of English language learners, the results demonstrate that these opportunities were provided to a diverse group of students, 
including traditionally underserved subgroups of students. 

3. The effects of attending a participating network school on deeper learning opportunities were similarly positive for initially 
high- and low-achievers and for students who did and did not qualify for free or reduced price lunch. 

4. Teachers’ most challenging assignments collected from the network schools exhibited greater opportunities for independent 
learning in mathematics and for real-world connections in English language arts than those collected from the non-network 
schools, but were not significantly different on other opportunity measures (including complex problem solving, communication, 
and conceptual understanding of mathematics). 

5. The opportunities for deeper learning experienced by individual students, regardless of the school they attended, were associated 
with those students' deeper learning outcomes. 
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Introduction 


Today’s high school students face a very different world than previous generations. With the rapid 
evolution of technology, the global expansion of jobs and businesses, and an ever more complex 
and diverse democracy, new graduates must navigate an environment that is rapidly and continually 
changing. Postsecondary institutions and business leaders have called for students to be better 
prepared for this environment, but despite decades of educational reform efforts, concerns 
continue that too few American students acquire the complex knowledge and skills required to 
become engaged and productive citizens of this changing world (Murnane & Levy, 1996; Levy & 
Murnane, 2013). These concerns are more pronounced in schools that serve disproportionate 
numbers of disadvantaged students. In response, a movement in support of “deeper learning” 
has emerged among researchers, policymakers, and practitioners in an effort to improve students’ 
future success in college and in their careers and civic life. 

The concept of deeper learning has been used both to describe a set of competencies or goals for 
students, and to characterize a way of learning (or a process) that promotes these competencies. 
The William and Flora Hewlett Foundation — a leader in the national initiative to promote deeper 
learning in schools — has defined deeper learning as “a set of competencies students must 
master in order to develop a keen understanding of academic content and apply their knowledge 
to problems in the classroom and on the job” (William and Flora Hewlett Foundation, 2013, p. 1). 
In this view, deeper learning focuses on the development of six interconnected competencies that 
many argue are prerequisites for success in college, career, and civic life: 

• Mastery of core academic content 

• Critical thinking and complex problem-solving skills 

• Effective communication skills 

• Collaboration skills 

• An understanding of how to learn 

• Academic mindsets (William and Flora Hewlett Foundation, 2013; Chow, 2010; Trilling, 2010). 

Taking a slightly different approach, a recent review of theory and research across an array of 
disciplines led a National Research Council panel (National Research Council [NRC], 2012) to 
define deeper learning as “the process through which an individual becomes capable of taking 
what was learned in one situation and applying it to new situations (i.e., transfer).” The panel 
distinguished that process from the specific “21st century competencies” it produces. The NRC 
grouped these competencies into three domains: the cognitive domain, the interpersonal domain, 
and the intrapersonal domain. These domains neatly subsume the six dimensions identified 
by the Hewlett Foundation, providing a compatible framework for the purposes of both research 
and practice. 
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The concept of deeper learning has gained momentum among educators and policymakers as a 
means to better prepare students for college and careers. However, the research on deeper 
learning has lagged behind the political and educational interest in this concept and the activity 
of practitioners in schools and districts. Although early evaluation studies of schools participating 
in deeper learning-focused networks suggested positive effects, they had a number of limitations 
(relating to their research designs, samples, data, measures, or analyses; Yuan & Le, 2010). More 
recent evaluations (Collins et al. , 2013; Guha et al . , 2014; Nichols-Barrer & Haimson, 2013) 
indicated positive program effects on indicators such as GPA, progress to graduation, or state test 
results. However, these studies represent a modest empirical research base on deeper learning, 
given that they are either primarily descriptive in nature or have focused on demonstrating the 
effectiveness of specific instructional programs or approaches aligned with the goals of deeper 
learning. Indeed, the NRC panel noted the limitations of existing (and primarily correlational) 
research in establishing linkages between 21st Century/deeper learning competencies and 
long-term outcomes for students, recommending that foundations and federal agencies support 
further research in this arena (NRC, 2012). As a result of this limited empirical base, there has 
recently been increased interest in rigorous research that evaluates whether school approaches 
explicitly focused on developing deeper learning competencies are associated with improved 
educational experiences and outcomes for students from all backgrounds. 1 

The Study of Deeper Learning: Opportunities and Outcomes — funded by the Hewlett Foundation — 
aimed to determine whether students who attended high schools with a mature and at least 
moderately well implemented approach to promoting deeper learning actually experienced greater 
deeper learning opportunities and outcomes than likely would have been the case had they not 
attended these schools. 2 In contrast to an evaluation of a particular program or instructional 
strategy, this proof-of-concept study focused on providing evidence about whether schools can 
promote deeper learning, across a variety of reasonably well implemented approaches and a 
diversity of students. This study aimed to address the evidence gap related to deeper learning 
by using a rigorous quasi-experimental design 3 to examine a set of high schools (hereafter 
referred to as “network” schools) associated with ten established networks from across the 
country that embrace the goals of deeper learning, promote instructional practices they believe 
are likely to lead to deeper learning competencies, and participate in the Hewlett Foundation’s 
Deeper Learning Community of Practice. (See Box 1 for a list of participating networks.) 


1 Prior research also notes the variation in classroom implementation of instructional approaches associated with reform efforts 
within and across schools, emphasizing the importance of examining opportunities to learn experienced by students at schools 
focused on deeper learning (e.g., see NCCSR, 2004; Berends et al., 2000; Aladjem et al., 2006). 

2 See our first report (Huberman et. al, 2014) for a description of the approaches to promoting deeper learning taken by schools 
in this study. 

3 A quasi-experimental design estimates the effect of a “treatment,” program, or intervention by comparing outcomes for people 
who chose or were selected to participate and those who did not, rather than by randomly assigning participants (see Shadish, 
Cook & Campbell, 2002). 
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As described in the first report in this series — The Shape of Deeper Learning: Strategies, Structures, 
and Cultures in Deeper Learning Network High Schools — the network schools included in this study 
shared an explicit, school-wide focus on deeper learning as a goal for students (Huberman, Bitter, 
Anthony, & O’ Day, 2014). While employing a diverse range of approaches to promote deeper 
learning, they had several strategies and structures in common. 

Box 1: Networks Participating in the Hewlett Foundation’s Deeper Learning Community of Practice 

Asia Society - http://asiasociety.org/international-studies-schools-network 
Big Picture Learning - http://www.bigpicture.org/ 

ConnectEd - http://www.connectedcalifornia.org/ 

EdVisions Schools - http://www.edvisions.com/ 

Envision Schools - http://www.envisionschools.org/ 

Expeditionary Learning - http://elschools.org/ 

High Tech High - http://www.hightechhigh.org/ 

Internationals Network for Public Schools - http://internationalsnps.org/ 

New Tech Network - http://www.newtechnetwork.org/ 

New Visions for Public Schools - http://www.newvisions.org/ 

Note: See our first report (Huberman et at, 2014) for more information on the Deeper Learning Community of Practice. 

The Hewlett Foundation’s deeper learning initiative reflects a set of core assumptions about how 
a school’s approach to promoting deeper learning can result in improved outcomes for students, 
including postsecondary success and civic and employment outcomes. In this study, we examined 
the underlying premise of this initiative: that explicit strategies, structures, and school culture 
designed to support deeper learning would result in students experiencing greater opportunities to 
engage in deeper learning. These opportunities would, in turn, lead to transferable knowledge and 
skills — or competencies — that are critical to readiness for college, career, and civic life. 

For analysis and interpretation, we group these competencies into three overlapping domains, 
as defined by the National Research Council (2012): the cognitive domain, including mastery of 
academic content knowledge and complex problem solving; the interpersonal domain, including 
collaboration and communication skills; and the intrapersonal domain, including an understanding 
of how to learn and academic mindsets, such as motivation to learn, academic engagement, and 
self-efficacy (Farrington et al. , 2012; Soland, Hamilton, & Stecher, 2013; NRC, 2012). Proponents 
of deeper learning argue that approaches focused on developing these competencies can improve 
outcomes for all students, including those from traditionally underserved groups and those who 
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have not previously experienced educational success. The abbreviated theory of action for the 
deeper learning initiative (shown in Exhibit 1) delineates the key hypothesized relationships 
between school approaches to promoting deeper learning, opportunities to engage in deeper 
learning, and outcomes. In this graphic, we provide additional detail related to the focus of this 
report — students’ experienced opportunities to engage in deeper learning. 

Exhibit 1: Abbreviated Theory of Action 4 



4 Not all of the outcomes included in this diagram are measured through survey and assessment data in this study (for example, 
understanding how to learn). Many of the intrapersonal outcomes shown in the diagram align with the sixth deeper learning 
competency: the development of academic mindsets. 
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Our first report from this study described key aspects of participating schools' approaches to 
promoting deeper learning (Huberman et al. , 2014) — that is what adults in network schools did to 
develop deeper learning competencies. In that report, we described the strategies, structures, and 
cultures network schools instituted to foster the development of the three competency domains 
(cognitive, interpersonal, and intrapersonal; see Box 2). In this report — our second in a series of 
three — we focus specifically on the opportunities that students experienced to engage in deeper 
learning in their classrooms. In other words, we address the following key question: 

Did students who attended the selected network schools experience more opportunities 
to engage in deeper learning than would likely have been the case had they not attended 
the network schools? 

This question addresses a fundamental assumption that underpins the deeper learning initiative: 
that students in schools with at least moderately implemented approaches to promoting deeper 
learning actually experience greater opportunities to engage in deeper learning across their 
classes. 5 This question is critical in that it asks whether students’ classroom experiences in the 
network schools are significantly different than what comparable students experience in other 
schools. It lays the foundation for an examination of the outcomes of deeper learning, since if we 
do not find a significant, meaningful difference in students' opportunities, then any differences in 
student outcomes that we may find would likely be due to other factors beyond students' classroom 
engagement in deeper learning. 

In addition, while one might expect that a school that focuses on deeper learning would provide 
deeper learning opportunities, prior educational research has shown that classroom implementation 
of an instructional approach or strategy can vary considerably within and across schools (NCCSR, 
2004; Berends et al., 2000; Aladjem et al., 2006). Therefore we cannot assume that schools 
implementing approaches to deeper learning are actually changing students’ learning experiences 
in a meaningful way, and we must determine whether there is evidence of such differences. 


5 Since this was a proof-of-concept study, our aim was to determine whether well implemented approaches to deeper learning 
resulted in greater opportunities and outcomes for students. We therefore asked network leaders to identify schools that were 
implementing the network’s approach to deeper learning at a moderate or high level based on their criteria as potential participants 
in the study. We did not include low implementers because the study aimed to determine whether the deeper learning approach, 
when at least moderately implemented, could result in greater opportunities and outcomes. 
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Box 2: Approaches and Structures for Deeper Learning 


In the first report of the Study of Deeper Learning (see Huberman et al., 2014) we described the approaches 
(including strategies, structures, and culture) that network schools took to promote deeper learning. The 
analysis drew on qualitative data and teacher survey data from a broader sample of 19 network high schools 
and 1 1 comparison high schools. Key findings from this report included: 

1. Across the three deeper learning domains, sampled network schools used a range of strategies to 
develop deeper learning competencies-most commonly project-based learning, internship 
opportunities, collaborative group work, and longer-term cumulative assessments. 

• In the cognitive domain, all but one network school (18 of 19) employed project-based learning (PBL) 
to some degree to develop mastery of core academic content knowledge and critical thinking skills. 
PBL was integral to daily instruction in slightly over a third of these schools and used more sporadically 
in others. 

• Also in the cognitive domain, three-quarters of the network schools (14 of 19) provided connections 
to the real world through internship opportunities for students. At two schools, internships were 
considered central to learning and occurred two to three days per week across all four years. The 
remaining 12 schools incorporated internships for a portion of students at some point in their school 
career to provide career-related experience, boost life skills, or help with the transition from high school 
to college and careers. 

• In the interpersonal domain, collaboration and communication skill development was an explicit 
goal reported by staff at a majority of network schools (11 of 19), which they addressed through 
collaborative group work and longer term assessments (such as portfolios and exhibitions, where 
students had to present and defend their work). 

• In the intrapersonal domain, almost half of the network schools (9 of 19) reported having explicit goals 
related to intrapersonal competencies (learning how to learn and academic mindsets) for students and 
they used a variety of strategies to encourage the development of these skills, including study groups 
and student participation in decision making. Three schools focused on individualized learning as a way 
to develop independent learning and self-management skills. 

2. Most network schools supported the implementation of instructional approaches aligned with 
deeper learning through the development of specific structural and cultural elements, including 
advisory classes (16 schools), alternative scheduling (14 schools) and personalized learning environments 
(all schools). However, these structures and cultures looked different across the schools. For example, 
advisory classes had different numbers of students (from 15 to 30 students), ran for different amounts of 
time (between 30 and 60 minutes), and happened with different frequencies (from every day to once or 
twice a week), depending on the school. 

3. Comparisons between the network and non-network school principal interview data suggest that 
the network schools employed strategies to foster the deeper learning competencies to a greater 
extent than did the non-network schools, particularly in the areas of project-based learning, internship 
opportunities, collaborative group work, longer-term cumulative assessments, and development of 
intrapersonal skills. Network schools also employed advisory classes and alternative scheduling to a 
higher degree than the non-network schools. 
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Study Design 

To examine the extent to which network schools provided more opportunities for students to 
engage in deeper learning, we surveyed students in these schools about the nature of their 
learning experiences, focusing on activities aligned with the six deeper learning competencies. 

We compared these experiences — or opportunities — with those reported by similar students 
attending other schools in the same geographic area that did not participate in one of the 10 deeper 
learning networks (hereafter referred to as “non-network schools”). We also examined whether 
certain subgroups of students (e.g., students with low and high levels of prior achievement) 
differentially benefited from attending a network school. In addition, we collected examples of 
the work teachers assigned to students (teacher assignments) in English Language Arts (ELA) 
and mathematics from both network and non-network schools to compare the opportunities 
for deeper learning provided by these assignments across the two groups of schools. 

Study Participants: This analysis includes 11 pairs of matched network and non-network schools 
in California and New York. We included network schools that were considered to be moderate or 
high implementers of one of the 10 network models (based on network representatives' reports), 
and we matched these schools with non-network schools that had similar incoming student 
populations (based on the demographics and achievements of incoming ninth-grade students). 

All schools were nonselective and served substantially disadvantaged populations. (See Box 3 
for additional details regarding the selection process for schools and students.) 

Measures of Opportunity: While we collected a wide range of data for this study, we relied primarily 
on student survey data and examples of assignments students were given by their teachers (teacher 
assignments) for these analyses (see Box 4). The survey included items that addressed nine 
measures of opportunities for deeper learning. These measures directly addressed four of the 
six dimensions of deeper learning outlined by the Hewlett Foundation (opportunities for complex 
problem solving, opportunities to collaborate, opportunities to communicate, and opportunities 
to learn how to learn), as well as additional opportunities that we expected would support the 
development of deeper learning competencies (opportunities for creative thinking, opportunities 
to receive feedback, opportunities for assessments aligned with deeper learning, opportunities 
for interdisciplinary learning, and opportunities for real world connections). 6 We measured the level 
of opportunity by asking students to report the total number of core academic classes (including 
English, mathematics, science, or social studies) in which they engaged in specific activities related 
to each of the identified opportunities in the current school year. 7 (See Box 5 for the opportunity 
measures we used, and Technical Appendix, Section III.A. for the actual survey items.) 


6 We did not measure opportunities to master core academic content or develop academic mindsets directly through the survey. 
However, opportunities to develop a conceptual understanding of core mathematics content were measured through the analysis 
of teacher assignments. Also, the opportunity measures from the student survey included only opportunities offered through 
core academic coursework (i.e. , they did not measure opportunities in elective classes or extracurricular activities). In addition, 
opportunities to learn how to learn and opportunities for real-world connections may reflect activities that promote the development 
of academic mindsets. 

7 While we relied in large part on validated survey items and scales to measure the multiple facets of deeper learning, the 
measurement scales capture some of these constructs in only a limited way. For example, the survey scales for opportunities 
focused on the breadth of opportunities across academic classes, rather than the precise frequency or quality of those opportunities. 
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Box 3: Participating Schools and Students 


The sample of schools included high schools participating in one of the 10 networks in the Hewlett Foundation’s 
Deeper Learning Community of Practice and identified as moderate or high implementers of their network's 
approach (“network schools"). In addition, the sample included a set of comparison non-network schools 
serving similar student populations but not participating in any of the 10 networks (“non-network schools”). 
While the non-network schools were not members of the 10 networks, and deeper learning was not necessarily 
a focus at the schools, they may have been implementing other reforms. 

Selecting Schools: We used several criteria to choose high schools from the participating networks. We 
selected schools that had implemented the network approach schoolwide and were considered to be 
moderate or high implementers of the approach, according to network representatives (based on criteria 
established by each network), we sought schools that had been in existence for at least four years (i.e., long 
enough to have graduated at least one class by the start of the study); that were non-selective in admissions 
(increasing the validity of comparisons between students in network and non-network schools); and that 
enrolled at least 200 students (ensuring a sufficiently large sample of students for data analyses). Because 
we were particularly interested in outcomes for economically disadvantaged students, we sought schools in 
which at least 25 percent of students were eligible for free or reduced-price lunch, although most of the schools 
we selected had a substantially larger percentage of eligible students (see Technical Appendix, Section II. A.). 

We identified a matched non-network comparison school for each network school using data from the 
Common Core of Data (CCD), as well as aggregated student-level data obtained from each district. Student- 
level district data, which we aggregated to the school level, included student demographic and socioeconomic 
characteristics, as well as incoming student test scores (i.e., Grade 8 mathematics and English Language Arts 
test scores). For each network school, we recruited a non-network school in the same district or in a neighboring 
district. We sought non-network schools that, like the network schools, had been in existence for at least four 
years, were non-selective in admissions, and enrolled at least 200 students, at least 25 percent of whom 
were eligible for free or reduced-price lunch. 

For the analysis described in this report, we included 11 pairs of schools with sufficient numbers of students 
who consented to participate in the study activities. The network schools represented 8 of the 10 networks 
and were located in five different districts across two states: California and New York. The non-network schools 
were located in six districts across these two states.* 

Selecting Students: Students who participated in the data collection activities had been in the district since 
Grade 8 (allowing us to measure prior levels of achievement and demographic characteristics) and who 
entered Grade 9 in one of the sampled schools between the 2009-10 and 2011-12 academic years. In 
addition, for the student survey, the sample was restricted to students with parental consent to participate 
in the study. In some cases, challenges associated with obtaining active consent (required by four districts) 
limited the number of identified students who could participate in data collection activities. We used several 
approaches to maximize the number of students included in the data collection activities and analyses, and 
we accounted for non-response in the analysis model (see Technical Appendix, Section IV.A.). 


* We included 12 network schools and 10 non-network schools. One non-network school was matched to two network schools, and 
two of the network schools were combined for analyses-because they had small student populations eligible for the study and were 
co-located on one campus-resulting in 1 1 parallel analyses. An additional eight network schools (including three schools in the 
two networks not included in this analysis) participated in the study but were excluded from this analysis, either because we did 
not identify a matched comparison school that met our criteria and was willing to participate, or because we were unable to obtain 
parental consent for a sufficient number of students for certain data collections. 
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Box 4: Data Sources 


The analyses of opportunities for deeper learning presented in this report rely on three primary data sources: 

1. A Student Survey: We administered a one-hour survey to participating students in spring 2013. The survey 
was designed to measure: 1) the opportunities students experienced in school related to the deeper 
learning competencies, and 2) the interpersonal and intrapersonal outcomes that are hypothesized to 
be important to college and career readiness in the theory of action. We administered the survey to all 
sampled and consented students in Grade 11 and Grade 12 in each participating network and non-network 
school. In total, we administered surveys for 1,762 students in 11 pairs of schools, with an average 
response rate of 76 percent among sampled students (ranging from 54 percent to 93 percent of sampled 
students, by school).* 

The survey included previously validated item sets from national surveys, including the Consortium of 
Chicago School Research (CCSR) survey and the High School Longitudinal Survey (see Technical Appendix, 
Section III.A. for a full list of items and sources). We supplemented these existing items with original items 
designed to address specific constructs important to this study. The survey was piloted prior to administration 
to test the validity of the scales. We calculated scale scores for the survey constructs using a Rasch 
analysis.** These Rasch scores were standardized using the mean and standard deviation among 
students from non-network schools so that the results could be presented as effect sizes. 

2. Teacher Assignments: We collected sample ELA and mathematics classroom assignments from teachers 
of a sub-sample of students in the 11 pairs of schools. The purpose of these data was to measure the 
most challenging opportunities students were given to demonstrate critical thinking, communication, and 
independent learning skills (“learning how to learn”) and to make real-world connections. While the survey 
gave us information about the usual instruction in the schools, the assignments gave us information about 
how students were challenged to go beyond typical expectations. We collected assignments from teachers 
of up to 40 randomly selected students per school (20 students in Grade 10 and 20 students in Grade 
12) in spring 2013. We asked each teacher to submit an example of the “most challenging assignment” 
given to the sampled students they were teaching, plus an additional challenging assignment. In total we 
collected 148 mathematics assignments and 148 ELA assignments. 

3. Extant Student-Level Data: We obtained extant student-level demographic data from the districts 
associated with the participating schools for five cohorts of students. We used these data to calculate 
propensity scores for students and to select student samples. Demographic data were also used as 
covariates in the analysis models. 


* A technological complication with online survey administration resulted in a large amount of missing data in one of the non-network 
schools. For survey constructs affected by this technological issue, analyses excluded this pair of schools. 

** Rasch analysis is a method of generating scale scores on a survey or test based on responses to individual items. 
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Box 5: Opportunities for Deeper Learning-Survey Measures 


1. Opportunities for complex problem solving: The degree to which students engage in complex problem 
solving by analyzing ideas, judging the value and reliability of an idea or source, constructing new ideas, 
and applying knowledge to solve new problems 

2. Opportunities for creative thinking: The extent to which students have the opportunity to engage in 
creative thinking in their core academic classes, such as thinking of original solutions to problems and 
new ways to do things, creating new ideas, and using their imagination. 

3. Opportunities to communicate: The extent to which students have the opportunity to practice written and 
oral communication skills 

4. Opportunities to collaborate: The degree to which students collaborate on assignments, provide feedback 
on each others’ work, and collaborate in other ways 

5. Opportunities to learn how to learn: The degree to which students practice monitoring and directing their 
own work and learning 

6. Opportunities to receive feedback: The degree to which students receive feedback on their work from 
teachers, peers, or others, and the form this feedback takes (written, oral, both) 

7. Assessments aligned with deeper learning: The extent to which students engage in various forms of 
assessment including assessments of problem solving, communication, and collaboration 

8. Opportunities for interdisciplinary learning: The degree to which students engage in interdisciplinary 
learning, where two or more disciplines are combined to enhance inquiry and knowledge generation 

9. Opportunities for real-world connections: The degree to which students engage in instructional activities 
that emphasize real-world connections 

Students were asked to respond to a set of items asking about the number of core content classes 
(including English, mathematics, science, and social studies) in which they engaged in activities relevant to the 
opportunity measure. Responses options included: 0 = none of my classes; 1 = one of my classes; 2 = two of 
my classes; 3 = three or more of my classes.* 

Analyses for opportunities for complex problem solving, opportunities to communicate, and opportunities 
to learn how to learn included 10 school pairs, while analyses of the remaining opportunities included 
11 school pairs. 


* Opportunities for interdisciplinary learning were measured on the response scale 0 = never, 1 = some of the time, 

2 = most of the time, 3 = all of the time. 

We also collected teachers’ most challenging assignments in ELA and mathematics in order to 
have an authentic measure of the most challenging types of instructional activities in which students 
were engaged at both network and non-network schools. ELA and mathematics experts scored 
these assignments using rubrics that we developed in consultation with the scorers. These rubrics 
had been validated in previous studies conducted by American Institutes for Research (AIR), and 
they were modified specifically to address several of the deeper learning competencies examined 
in this study. The scorers used the rubrics to rate the quality of each assignment on either four 
criteria (ELA assignments) or five criteria (mathematics assignments), including: conceptual 
understanding of core content (mathematics only); critical thinking skills; effective communication 
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skills; independent learning skills (e.g., learning how to learn); and real-world connections 
(see Box 7 in Key Questions and Findings for scoring criteria). Additional information on the 
scoring rubrics is presented in the Technical Appendix, Section III.C. 8 In addition to looking at the 
individual scoring criteria, we estimated an overall score for each assignment. 

The teacher assignments and student survey data provided complementary information on 
opportunities for deeper learning. The goal of the survey was to collect data directly from a large 
sample of students in Grades 11 and 12 on the opportunities they experienced in instruction 
across all core classes. The collection of assignments, meanwhile, focused on a smaller sample 
of students in order to provide an exploratory analysis of authentic opportunity data in ELA and 
mathematics classes only. In addition, while the survey asked students to report on their 
classwork in general, the teacher assignments identified opportunities provided through the 
most challenging assignments that students received, thus representing only a specific subset 
of instructional experiences. The analysis models used for both of these data sources are 
summarized in Box 6. 

Box 6: Analysis Models 

Analysis of Survey Measures of Opportunities in Network and Non-Network Schools: To estimate the 
differences in opportunities for deeper learning reported by students in network and non-network schools, we 
used doubly robust ordinary least squares regression models.* We balanced student characteristics within 
pairs of schools using propensity score weighting** (described in the Technical Appendix, Section IV.A.), 
and accounted for non-consent and non-response. We performed student-level analyses separately within 
each pair of network and non-network schools, using the combined sample of students who entered Grade 9 
in the 2009-10 and 2010-11 academic years, taking student cohort and characteristics measured prior 
to entry into high school into account. We then used a meta-analytic technique to estimate the average 
difference between students in network and non-network schools across the 11 pairs of schools. (See the 
Technical Appendix, Section IV.B. for a more detailed description of the analysis method.) 

Analysis of Teacher Assignment Measures of Opportunities in Network and Non-Network Schools: 

We used a two-level hierarchical linear model (HLM) to estimate differences in scores between the network 
and non-network assignments (for each scoring criterion and overall). Assignments were weighted by the number 
of sampled students to which they were assigned, and grade level was taken into account in the model. 


* The analysis is considered doubly-robust (Funk et at, 2011) because it accounts for preexisting differences between network and 
non-network students in two ways: first by adjusting for how the student differences are associated with network school selection 
(using propensity score weighting), and second, by adjusting for how the student differences are associated with opportunity measures 
(using regression models). If either of the two adjustment methods accurately accounts for student differences, we can obtain valid 
estimates of the network school’s effect. 

** Propensity score weighting is an analysis technique that balances the sample of students attending program and comparison schools 
to ensure that comparisons are made among similar students. 


The scoring rubrics are also available upon request. 
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Key Questions and Findings 

In this report, we explore the extent to which students attending network schools experienced 
opportunities for deeper learning, and whether they experienced more opportunities to engage in 
deeper learning than would likely have been the case if they attended a non-network school. In this 
section, we summarize findings relating to five key questions. 


To What Extent Did Students in Network High Schools Experience 
Opportunities for Deeper Learning? 

Students in network schools reported experiencing opportunities to develop certain deeper 
learning competencies (e.g., collaboration and learning-to-learn) more than others. On the 

survey, students were asked the number of core content classes (on a scale of 0 to 3 where 3 = 
three or more classes) in which they participated in activities aligned with the deeper learning 
competencies. Students in network schools reported experiencing opportunities to learn how to 
learn and to collaborate in more than two core content classes on average (mean score = 2.48 
and 2.41 respectively). However, they reported opportunities for complex problem solving less 
often — on average, in one to two classes (mean score = 1.81). (See Exhibit 2 for the average 
score across network schools for all opportunity measures.) 9 These data indicate that network 
schools may have focused on developing certain dimensions of deeper learning more than others, 
or that certain competencies were developed in a broader range of classes. Our first report in this 
series (Huberman et al . , 2014) provides examples of the approaches used to provide these 
opportunities and provides insight into this variation. 


Exhibit 2: Average Opportunity to Learn Across All Network Schools; Scale = 0 Classes to 3 or More Classes 


Opportunities for complex 
problem solving 

Opportunities for creative thinking 

Opportunities to communicate 

Opportunities to collaborate 

Opportunities to learn how to learn 

Opportunities to receive feedback 
Assessments aligned with 
deeper learning 
Opportunities for real 
world connections 


1.81 


1.98 


2.15 


41 

2.48 


2.17 

2.18 


1.97 


0.0 0.5 1.0 1.5 2.0 2.5 3.0 


Number of Classes (3=three or more classes) 


Note: The bars represent the estimated mean of the school means for each opportunity scale, translated to the original response scale 
(0 = no classes, 1 = one class, 2 = two classes, 3 = three or more classes). 


9 Opportunities for interdisciplinary learning were measured on the response scale O = never, 1 = some of the time, 
2 = most of the time, 3 = all of the time. The mean score for this measure was 1.56. 
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We observed similar variation in opportunities across competencies both in the network and 
non-network schools, but how did students’ opportunities for deeper learning differ between 
network and non-network schools? The following results focus on the differences in the opportunities 
experienced in network and non-network schools overall and within individual pairs of schools. 

Did Students Experience More Opportunities for Deeper Learning in 
Network Schools Than in Non-Network Schools? 

Students in participating network high schools reported experiencing significantly more 
opportunities for deeper learning than their matched counterparts in the paired non-network 
schools. These significant differences held true for all opportunity measures. We measured these 
differences in terms of effect sizes, which represent the difference between the two groups of 
students in standard deviation units. The average effect sizes ranged from 0.21 (opportunities for 
assessments aligned with deeper learning) up to 0.55 (opportunities to collaborate). The average 
effect sizes across all pairs of schools for all nine opportunity measures are shown in Exhibit 3. 
The vertical lines represent the 95 percent confidence intervals. Because the confidence intervals 
all lie above the zero line, they demonstrate that all measures are statistically significant and 
positive. We observed positive effects even on the opportunity measures that were relatively low 
for the network schools, as described above. 

To more easily interpret the magnitude of these differences, we translated the effect sizes to the 
original survey scale. For example, the effect size of 0.55 standard deviations on the Rasch scale 
for opportunities to collaborate translates to 0.5 points on the original 0 to 3 survey scale 
representing the number of core content classes (where 3 = three or more classes) in which they 
participated in activities aligned with the deeper learning competencies. That is, if, on average, 
students in non-network schools experienced these opportunities in two of their core courses, 
their counterparts in network schools more often reported opportunities in three or more of their 
core courses. In the Technical Appendix, Section V.A., we provide translations to the original survey 
scale for each opportunity measure. 
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Exhibit 3: Estimated Average Effect of Attending a Network School on Students’ Opportunities for Deeper Learning 
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Note: The plotted points represent the meta-analytic average effect estimate for each opportunity measure (see Technical Appendix IV.B. 
for statistical model used), and the vertical bars represent each estimate’s 95 percent confidence interval. Effect sizes are significant and 
positive when the full confidence interval lies above the zero line (all measures in this chart). 

Although the meta-analytic results averaging across all 11 pairs of schools are the most precise, 
our analysis model also allowed us to identify significant differences in reports of opportunities 
within individual pairs of schools. This analysis revealed that for each opportunity measure, there 
was variation in effect of network attendance by school pair. In other words, some network 
schools may have implemented approaches to promote a given deeper learning competency more 
comprehensively than other network schools did. For example, in eight pairs of schools, student 
reports of opportunities for collaboration were significantly higher in the network schools than in 
the non-network schools. Among these pairs, the effect sizes ranged from 0.45 up to 1.19. For 
three pairs, however, there was no significant effect for this measure (see Exhibit 4). 10 As another 
example, with respect to opportunities for complex problem solving (Exhibit 5), student reports of 
opportunities were significantly higher in the network schools than the non-network schools for 
four pairs (effect sizes among these pairs ranged from 0.35 to 0.66). For six network schools, we 
observed no significant effect for this measure. In some cases, these insignificant effects may 
have been due to small sample sizes within pairs. Charts showing effect sizes by pair for each 
opportunity measure are included in Technical Appendix, Section V.A. These data suggest that for 
each opportunity measure, some network schools were more successful in providing increased 
opportunities than other network schools. 


10 In some cases these insignificant results may be due to small sample sizes within pairs. 
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Exhibit 4: Estimated Effect of Attending a Network School on Students’ Opportunities for Collaboration, 
by Individual School Pair 
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Note: The plotted points represent the effect estimate for opportunity to collaborate for each school pair included in the meta-analysis. 
The effect estimate for each pair on the Rasch scale is provided directly under each plotted point. The top scale represents the effect size 
based on the Rasch scale (in standard deviations); the bottom scale shows the effect size translated into the original response scale 
(O = no classes, 1 = one class, 2 = two classes, 3 = three or more classes). The horizontal bars represent each estimate’s 95 percent 
confidence interval. Effect sizes are significant and positive when the full confidence interval lies to the right of the zero line. 
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Exhibit 5: Estimated Effect of Attending a Network School on Students’ Opportunities for Complex Problem Solving , 
by Individual School Pair 
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Note: The plotted points represent the effect estimate for opportunity for complex problem solving for each school pair included in the 
meta-analysis. The plot also includes Pair 07 (grey point) for reference, which was not included in the meta-analysis because of a low 
response rate for the opportunity for complex problem solving survey items. The effect estimate for each pair on the Rasch scale is provided 
directly under each plotted point. The top scale represents the effect size based on the Rasch scale (in standard deviations); the bottom 
scale shows the effect size translated into the original response scale (0 = no classes, 1 = one class, 2 = two classes, 3 = three or more 
classes). The horizontal bars represent each estimate’s 95 percent confidence interval. Effect sizes are significant and positive when the 
full confidence interval lies to the right of the zero line. 


In addition, we found that for each network school, there was variation in effect by opportunity 
measure. In other words, individual network schools were more successful in providing greater 
opportunities for certain dimensions of deeper learning than they were for other dimensions. 

For example, students at one network school reported experiencing greater opportunities for 
collaboration than students in the paired non-network school (with an effect size of 0.74) but did 
not demonstrate significant positive effects for any other opportunity measure (see Exhibit 6 for 
“Pair 5” results). The network school students within another pair reported greater opportunities 
for deeper learning across all measures (see Exhibit 7 for “Pair 6” results), but with varying effect 
sizes. In 3 of the 11 pairs, significant positive differences were observed for only one opportunity 
measure, while in eight pairs, significant positive differences were observed for multiple opportunity 
measures (see Technical Appendix, Section V.A. for detailed pair results). 
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Exhibit 6: Estimated Effect of Attending a Network School on Students’ Opportunities for Deeper Learning, 
Pair 5 Results 
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Note: The plotted points represent the effect estimate for each opportunity measure for Pair 5. The vertical bars represent each estimate's 
95 percent confidence interval. Effect sizes are significant and positive when the full confidence interval lies above the zero line. 


Exhibit 7: Estimated Effect of Attending a Network School on Students’ Opportunities for Deeper Learning, 
Pair 6 Results 


Effect Size 
1.50 


Collaboration 


Interdisc. 

Learning 


1.25 

1.00 

0.75 

0.50 

0.25 

0.00 


Complex 

Problem 

Solving 


0.66 


Creative 

Thinking 

Communication 


10.84 


10.78 


11.19 


Receive 

Feedback 


Learn How 
to Learn 


10.95 


Assessments 

Aligned 

^ 0,28 


11.14 


Real-World 

Connections 


10.58 


-0.25 


-0.50 


Note: The plotted points represent the effect estimate for each opportunity measure for Pair 6. The vertical bars represent each estimate's 
95 percent confidence interval. Effect sizes are significant when the full confidence interval lies above the zero line. 
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Despite the variation between and within schools in reported opportunities, students in each of 
the 11 network schools reported experiencing significantly more opportunity for deeper learning 
on at least one opportunity measure than students in the paired non-network schools. In addition, 
we observed no negative effects of network attendance on opportunities within any individual 
pairs. This consistency across pairs underscores the overall average positive effects associated 
with network participation. 

As we described in our first report, the study team also collected qualitative data (in addition to 
survey data), including interview and focus group data from principals, teachers, students, and 
network staff from the network schools. These data provide rich descriptions of the approaches 
network schools used to enable these opportunities for deeper learning and the variation in these 
approaches (see our first report, Huberman et al. , 2014). 

Were Effects of Participation on Opportunities for Deeper Learning Similar 
for All Students? 

The consistent average positive effects suggest that mature network schools were successful, 
on average, in providing greater opportunities for students to engage in deeper learning. These 
results, however, raise the question of whether all students in the network schools experienced 
these opportunities, or whether opportunities for deeper learning were differentially experienced by 
subgroups of students. For example, one might imagine that students with higher prior achievement 
may have enrolled in more advanced courses with more opportunities. To explore this issue, we 
looked at whether the effects differed among certain subgroups of students: students with low 
and high levels of prior achievement; students who did or did not qualify for free or reduced price 
lunch; students of different genders; and students in different grades. 11 Of these categories, only 
gender and grade level revealed differences in the effect of network participation on opportunities 
for deeper learning. 

Students With Low and High Levels of Prior Achievement 

Students with low prior achievement and students with high prior achievement both experienced, 
on average, greater opportunities for deeper learning than their counterparts in non-network 
schools 12 . In other words, we observed no difference in the effect of attending a participating 
network school on deeper learning opportunities between these two subgroups of students. 
Therefore, students with low and high levels of prior achievement appeared to experience the 
same benefits of network participation in terms of opportunities for deeper learning. Since 
high achievers traditionally are provided greater opportunities to learn through more advanced 
coursework and challenging instruction, this finding suggests that the network schools provided 
more equitable opportunities to students of all ability levels than is typical. 
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11 The distribution of students by race and ethnicity resulted in sample sizes that were considered too small for a valid analysis of 
racial/ethnic subgroups. 

12 Prior achievement was based on students' Grade 8 ELA scores. For California, we conducted the analysis by comparing students 
who entered high school with ELA test scores below and above the average for their state; for New York City, we compared 
students below and above average for their district. 
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Students Who Did or Did Not Qualify for Free or Reduced Price Lunch 


This analysis was limited to a smaller subgroup of six school pairs for which we were able to 
obtain free or reduced price lunch eligibility data. Once again, we observed no difference in the 
effect of network participation on the opportunities reported by the two subgroups of students. 

In other words, the positive effects of attending a network school were similar for students from 
low- and high-income families. 

Gender 

We also examined whether male and female students appeared to differentially benefit from 
attending a network school in terms of reported opportunities for deeper learning. On eight out 
of nine opportunity measures (the exception being interdisciplinary learning opportunities), 
we observed a larger positive effect of network participation on opportunities for deeper learning 
for females than males. For two measures — opportunities to learn how to learn and opportunities 
for real-world connections — there was no significant effect for males, but for females there was 
a positive, significant difference in opportunities for deeper learning between network and 
non-network schools. For the remaining six measures, students of both genders reported 
significantly more opportunities for deeper learning in network schools than non-network 
schools, but the effect sizes were larger for female students. 

Grade 11 and Grade 12 Students 

We also examined whether the benefits of attending a network school differed for Grade 11 and 12 
students. One might expect, for example, that certain activities or structures related to deeper 
learning opportunities (such as internships or senior projects) take place more commonly in 
Grade 12. We found larger positive effects of network participation for students in Grade 12 for 
some opportunity measures, including opportunities for creative thinking, opportunities to learn 
how to learn, opportunities to receive feedback, and opportunities for real-world connections. 
For all other measures, however, we did not observe any significant differences in effects between 
Grade 11 and Grade 12 students. 

Did the Most Challenging Assignments in Network High Schools Provide 
Greater Opportunities for Deeper LearningThan Those in Non-Network 
High Schools? 

Teacher assignment data complement the student survey data on opportunities for deeper learning 
by providing a glimpse into the types of assignments that students were actually asked to complete. 
Four competencies addressed in the assignment scoring rubrics (critical thinking skills, effective 
communication skills, independent learning skills, and real-world connections) overlapped with 
the competencies measured in the student survey. Flowever, we were able to assess these 
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competencies in a different way through the teacher assignments. While the survey used student 
reports to assess the prevalence of learning activities aligned with deeper learning, collecting and 
analyzing the most challenging assignments allowed us to assess the extent to which the most 
challenging ELA or mathematics assignments given to students during the school year aligned 
with high-quality deeper learning opportunities. In other words, the assignment collection did not 
measure the extent to which students experienced opportunities for deeper learning on a day-to- 
day basis or through their most typical assignments. Instead, it focused on determining whether 
challenging assignments at deeper learning schools offered opportunities to learn above and 
beyond the opportunities provided through challenging assignments at non-network schools. In 
addition, the mathematics assignment collection allowed us to measure opportunities to develop 
conceptual understanding of core content, which was not addressed in the survey. 

Box 7: Opportunities for Deeper Learning-Teacher Assignment Scoring Criteria (See Technical Appendix, 
Section III.C. for additional detail on scoring rubrics; scoring rubrics available upon request) 


Assignments were scored using a rubric that assessed the following dimensions. Assignments were scored on 

a scale of 1-3 or 1-4, depending on the criterion, as noted below. 

English Language Arts 

1. Critical thinking/creative thinking skills (1-3): The extent to which students are expected to demonstrate 
critical thinking skills in expository assignments by completing tasks that call for student work that moves 
beyond the reproduction of information to the construction of knowledge; or the extent to which students 
are expected to learn about literary elements of a genre and create a point in an imaginative writing 
assignment. Our analysis combines these measures for expository and imaginative assignments. 

2. Effective communication skills ( 1-4): The extent to which tasks call for student work that demonstrates 
effective, elaborated written communication for expository or imaginative written assignments 

3. Independent learning skills (1-4): The extent to which students partner with faculty in crafting tasks that 
meet students’ instructional goals 

4. Real-world connections (1-4): The extent to which teachers’ assignments ask students to connect the 
material they are studying with the world beyond school by thinking through how that material helps them 
address the problems and issues they must contend with in the real world 

Mathematics 

1. Conceptual understanding of core content (1-4): The extent to which teacher assignments focus 
primarily on asking students to demonstrate conceptual understanding as it relates to one or more core 
ideas in mathematics 

2. Critical thinking skills (1-4): The extent to which teacher assignments focus primarily on asking students 
to demonstrate critical thinking skills, such as problem solving and/or reasoning skills 

3. Effective communication skills (1-3): The extent to which communicating mathematical understanding 
is an explicit expectation of the assignment 

4. Independent learning skills (1-4): The extent to which students partner with faculty to develop 
mathematical tasks that meet students’ instructional goals 

5. Real-world connections (1-4): The extent to which teacher assignments address mathematical questions, 
issues, or problems similar to those likely to be encountered by professionals in the real world who use 
mathematics to solve problems 
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The percentage of most challenging ELA and mathematics assignments that were highly scored 
varied across the scoring criteria and indicated that there was room for improvement in the 
opportunities provided by classroom assignments (See Box 7 for scoring criteria). While we found 
examples of assignments that demonstrated high opportunities for deeper learning (see Technical 
Appendix, Section III.C. for examples of highly scored assignments), overall a low percentage of 
assignments in both the network and non-network schools received high scores across the 
measures (see Technical Appendix, Section V.C. for average scores of all criteria for students in 
network and non-network schools). In mathematics, only a small percentage of the most challenging 
assignments demonstrated a high level of opportunity on any given criterion. For example, only 
15 percent of the assignments that teachers gave sampled students in network schools demanded 
either “some” or “substantial” critical thinking (score of 3 or 4) within a complex, non-routine 
problem. Twenty-seven percent of network school mathematics assignments required a solution 
path and explanation, and explicitly set expectations for the presentation of the solution (score of 
3 on the communication criterion). ELA assignments received higher scores on average on some 
criteria. For example, 39 percent of network school assignments went substantially beyond the 
reproduction of information or required students to make a point in their writing (score of 3 on 
the critical thinking/creative thinking criterion), and 89 percent of network school assignments 
required extended writing (score of 3 or 4 on the communication criterion). However, few of the 
most challenging ELA assignments demonstrated a high level of opportunity for real-world 
connections or independent learning skills. 

We compared the scores of assignments from network and non-network schools and found that 

the mathematics assignments we collected in network schools were significantly more likely 
to exhibit opportunities for independent learning than the mathematics assignments from 
non-network schools, and the ELA assignments in network schools were significantly more 
likely to exhibit opportunities for real world connections than the ELA assignments from 
non-network schools. The independent learning measure for teacher assignments aligns with the 
deeper learning dimension “learning how to learn.” On all other measures for mathematics and 
ELA assignments, opportunity scores for the challenging assignments given to students 
in network and non-network schools were not significantly different. In other words, the 
challenging assignments given to both sets of students were similar in terms of the opportunities 
they provided students to demonstrate communication or critical thinking skills and demonstrate 
conceptual understanding of core content (mathematics only). 

In sum, the student survey data demonstrate that there is a greater prevalence of opportunities 
for deeper learning across core classes in the network schools than in the non-network schools. 
However, for most dimensions of deeper learning, the most challenging assignments were rated 
similarly at both network and non-network schools. How do we explain this apparent discrepancy? 
One plausible explanation is that students in the network schools were given such challenging 
assignments more frequently and in more of their classes than were students in the comparison 
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sites. Unfortunately our current data do not allow us to explore this hypothesis. This may be a 
useful topic for further research. 

Do Students Experiencing Greater Opportunities for Deeper Learning 
Also Experience Better Outcomes? 

Based on our survey measures of opportunities for deeper learning, we found that students 
attending network schools reported experiencing greater opportunities for deeper learning than 
students attending non-network schools. However, to what extent are these opportunities 
important for students' college and career readiness? While the average outcomes associated 
with attending deeper learning network schools are the focus of our third report, we began to 
address this question by considering whether individual students who reported receiving more 
opportunities for deeper learning — regardless of the school they attended — also demonstrated 
improved outcomes. The outcomes we considered included competencies associated with deeper 
learning, including students’ creative thinking skills, collaboration skills, academic engagement, 
motivation to learn, self-efficacy, locus of control, perseverance, and self-management. We also 
examined performance on an assessment of critical thinking and content knowledge (the 
Organisation for Economic Co-operation and Development [OECD] Programme for International 
Student Assessment [PISA]-Based Test for Schools). 

This analysis tests the assumption that greater opportunities for deeper learning are associated 
with improved student outcomes. The data and methods for this analysis are presented in Box 8. 
See Exhibit 8 for graphical examples of the general relationship between students’ opportunity 
measures and outcomes. 

Overall, we found a statistically significant relationship among each of the nine measures of 
opportunities for deeper learning and the eight measures of cognitive, interpersonal, and 
intrapersonal outcomes measured on the survey, for individual students (see Technical 
Appendix, Section V.D.). Each estimated relationship was statistically significant, with a 1.0 
standard deviation difference in an opportunity measure associated with a 0.12 to 0.52 standard 
deviation difference in an outcome measure (see Exhibits 9 and 10). For example, the estimated 
relationship between opportunities for complex problem solving and motivation to learn was 0.46 
among the California schools, suggesting that a student with opportunities for complex problem 
solving that were one standard deviation higher than average had a measure of motivation to 
learn that was 0.46 standard deviations higher than average. Similarly, the estimated relationship 
between opportunities for collaboration and academic engagement was 0.34 among the California 
schools, indicating that a student with opportunities to collaborate that were one standard deviation 
higher than average had a measure of academic engagement that was 0.34 standard deviations 
higher than average. 
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Box 8: Relating Opportunities to Outcomes 


To test whether the opportunities reported by students were related to outcomes, we examined the 
association-regardless of individual students’ participation in a network school-between opportunities 
(measured through the student survey) and key interpersonal, intrapersonal, and cognitive outcomes 
(measured through the survey and through the OECD PISA-ESased Test for Schools). 

Measurement of Interpersonal, Intrapersonal, and Cognitive Outcomes: 

Student survey: The student survey administered to students attending network and non-network schools 
in spring 2013 included measures of eight cognitive, interpersonal, and intrapersonal outcomes, including 
creative thinking skills, collaboration skills, academic engagement, motivation to learn, self-efficacy, locus 
of control, perseverance, and self-management. These measures were drawn or adapted from existing 
instruments (which had been piloted and scaled) and are based in the interpersonal and intrapersonal 
outcomes and academic mindsets literature cited earlier (e.g., Farrington et al., 2012; Soland, Hamilton 
& Stecher, 2013; National Research Council, 2012). 

OECD PISA-Based Test for Schools: We administered the OECD PISA-Based Test for Schools to examine 
two key deeper learning outcome dimensions: critical thinking and mastery of content knowledge. For the 
purposes of this research, we contracted with the approved U.S. vendor to administer the OECD PISA-Based 
Test for Schools to all sampled and consented students in Grades 11 and 12 in each network and non-network 
school in spring 2013. We administered the assessment to these grade levels, rather than the traditional age 
sample for the OECD PISA-Based Test for Schools (which typically includes students in Grade 9 and 10), 
because they had experienced a longer exposure to the network’s deeper learning approach.* In total, we 
administered the assessment to 570 students in network schools and 697 students in non-network schools, 
with an average response rate of 61 percent. The OECD PISA-Based Test for Schools is based on the 
internationally recognized PISA and provides measures of students’ content knowledge and application 
of higher-order thinking skills in reading, mathematics, and science. 

Analysis Method: We used a general three-level HLM model, adjusting for student-level Grade 8 characteristics, 
to identify the relationship between opportunities and outcomes. We conducted separate analyses for different 
regions (California and New York) because the available data on student characteristics differed across districts. 


* We also administered the assessment to the traditional age sample (15 year olds) in a subset of schools but did not use those 
data in the analysis because of the short time frame that those students had been in the schools. The results for the traditional 
sample were provided to the schools so they could see their students’ performance against established international benchmarks 
and performance patterns. 
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Exhibit 8: Two Examples of the Student-Level Relationship Between Opportunity Measures and Outcomes: 
California Schools 


Motivation to Learn 


Academic Engagement 




Opportunities for Complex Problem Solving Opportunities for Collaboration 


Note: Each plotted point represents a student's opportunity measure and outcome measure, expressed in standard deviation units. 
The diagonal line represents the general relationship between the opportunity and outcome measure. For display clarity, not all students 
are plotted. 


The relationship between the opportunities for deeper learning measures and OECD PISA-Based 
Test for Schools scores was less pronounced, however. Only our measure of opportunities for 
complex problem solving had a positive statistically significant relationship with OECD PISA- 
Based Test for Schools scores in mathematics and science. This relationship is one that might 
be expected because the OECD PISA-Based Test for Schools aims to measure problem solving and 
critical thinking skills. One would not necessarily expect a relationship between some of the other 
measures, such as opportunities for collaboration, and the individually administered OECD PISA- 
Based Test for Schools scores, which does not measure skills in collaboration. 
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Exhibit 9: Estimated Student-Level Relationship Between Opportunity Measures and Outcomes: California Schools 


Interpersonal and Intrapersonal Outcome Measures 


OECD PISA-Based Test 
for Schools 



Creative 

Thinking 

Collaboration 

Skills 

Academic 

Engagement 

Motivation 
to Learn 

Self- 

Efficacy 

Locus of 
Control 

Perseverance 

Self- 

Mgmt 

Reading 

Math 

Science 

Opportunities 
for Complex 
Problem Solving 

0.41 

0.43 

0.40 

0.46 

0.42 

0.37 

0.42 

0.39 

ns 

0.05 

0.08 

Opportunities 
for Creative 
Thinking 

0.36 

0.33 

0.33 

0.37 

0.36 

0.34 

0.34 

0.30 

ns 

ns 

ns 

Opportunities 

for 

Communication 

0.39 

0.33 

0.35 

0.41 

0.33 

0.35 

0.37 

0.35 

ns 

ns 

ns 

Opportunities 

for 

Collaboration 

0.25 

0.29 

0.34 

0.35 

0.29 

0.31 

0.30 

0.27 

ns 

ns 

ns 

Opportunities 
for Learning 
How to Learn 

0.24 

0.29 

0.35 

0.33 

0.27 

0.26 

0.29 

0.24 

ns 

ns 

ns 

Opportunities 
to Receive 
Feedback 

0.30 

0.35 

0.41 

0.42 

0.41 

0.37 

0.36 

0.32 

ns 

ns 

ns 

Opportunities 
for Assessments 
Aligned with 
Deeper 
Learning 

0.30 

0.30 

0.35 

0.39 

0.33 

0.35 

0.33 

0.31 

ns 

ns 

ns 

Opportunities 

for 

Interdisciplinary 

Learning 

0.27 

0.27 

0.24 

0.31 

0.27 

0.26 

0.24 

0.24 

ns 

ns 

ns 

Opportunities 
to Make 
Real-world 
Connections 

0.33 

0.38 

0.40 

0.45 

0.46 

0.44 

0.37 

0.36 

ns 

ns 

ns 


Note: The numbers in this exhibit represent the expected standard deviation difference in the outcome for a standard deviation difference 
in the OTL measure, based on an analysis model that accounts for measurement error in the outcome, student-level characteristics, and 
students nested in schools (see Technical Appendix, Section IV.B.). All coefficients shown are significant at a 0.05 level; non-significant 
results are designated by “ns.” 
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Exhibit 10: Estimated Student-Level Relationship Between Opportunity Measures and Outcomes: New York Schools 


OECD PISA-Based Test 

Interpersonal and Intrapersonal Outcome Measures for Schools 



Creative 

Collaboration 

Academic 

Motivation 

Self- 

Locus of 


Self- 





Thinking 

Skills 

Engagement 

to Learn 

Efficacy 

Control 

Perseverance 

Mgmt 

Reading 

Math 

Science 

Opportunities 
for Complex 
Problem Solving 

0.43 

0.51 

0.47 

0.52 

0.45 

0.43 

0.48 

0.45 

ns 

0.14 

0.12 

Opportunities 
for Creative 
Thinking 

0.29 

0.30 

0.31 

0.34 

0.38 

0.39 

0.33 

0.32 

ns 

ns 

ns 

Opportunities 

for 

Communication 

0.30 

0.25 

0.27 

0.30 

0.29 

0.30 

0.28 

0.31 

ns 

ns 

ns 

Opportunities 

for 

Collaboration 

0.29 

0.36 

0.32 

0.31 

0.31 

0.34 

0.27 

0.31 

ns 

ns 

ns 

Opportunities 
for Learning 
How to Learn 

0.18 

0.29 

0.35 

0.24 

0.25 

0.27 

0.18 

0.25 

ns 

ns 

ns 

Opportunities to 

Receive 

Feedback 

0.25 

0.33 

0.36 

0.34 

0.36 

0.37 

0.30 

0.32 

-0.11 

ns 

ns 

Opportunities 
for Assessments 
Aligned with 
Deeper 
Learning 

0.18 

0.24 

0.27 

0.24 

0.28 

0.32 

0.22 

0.25 

ns 

ns 

ns 

Opportunities 

for 

Interdisciplinary 

Learning 

0.16 

0.12 

0.12 

0.22 

0.30 

0.26 

0.13 

0.27 

ns 

ns 

ns 

Opportunities 
to Make 
Real-world 
Connections 

0.25 

0.30 

0.35 

0.36 

0.36 

0.36 

0.29 

0.34 

ns 

ns 

ns 


Note: The numbers in this exhibit represent the expected standard deviation difference in the outcome for a standard deviation difference 
in the OTL measure, based on an analysis model that accounts for measurement error in the outcome, student-level characteristics, and 
students nested in schools (see Technical Appendix, Section IV.B.). All coefficients shown are significant at a 0.05 level; non-significant 
results are designated by “ns.” 


26 


REPORT 2 | FINDINGS FROM THE STUDY OF DEEPER LEARNING Opportunities and Outcomes 






Box 9: Design Limitations 


While this study used a rigorous design with strict school and student selection criteria to ensure that we had 
sufficient numbers of students to make valid conclusions, we note a few limitations. First, because this was 
a proof-of-concept study rather than an evaluation of the network approaches, we only included schools that 
implemented the network approaches at a moderate or high level, which means that the findings cannot be 
generalized to all schools that are trying to implement approaches to deeper learning. Furthermore, the 
sample ultimately did not include some of the network schools that had implemented the models to the 
highest standard (due to the application of selection criteria relating to school size, grade range, or ongoing 
participation in other studies, for example) and it focused only on two specific state contexts. 

In addition, while the network schools were not academically selective in admissions, in some cases, students 
self-selected into the network schools. These students may have differed in some unmeasured ways from 
students with otherwise similar characteristics and prior performance who did not choose to attend a network 
school. Participants from network schools may also have been more invested in the study due to their interest 
in deeper learning, although we found no evidence suggesting that this was the case. Finally, despite having 
sufficient numbers of participating students and strong initial matching procedures, non-consent and 
non-response reduced the sample for some analyses. To adjust for this limitation, we took non-consent and 
non-response into account in the analysis (see Technical Appendix, Section IV.A.). 


Key Takeaways 

The findings outlined in this report provide a promising foundation for further exploration 
of the outcomes associated with approaches to promoting deeper learning. They support the key 
assumption underlying the Hewlett Foundation’s deeper learning initiative: that the network schools 
can provide greater opportunities for students to develop the deeper learning competencies, and 
that they can do so for students from a range of backgrounds and initial achievement levels. This 
finding is non-trivial. Prior research has shown that schools often struggle to implement programs 
and instructional initiatives across classrooms and for all students. This study demonstrates that 
it is possible to implement a deeper learning-focused instructional approach (in many different 
ways) that offers opportunities for a wide range of students. While we note several limitations 
of our study design in Box 9, the consistency of these findings across multiple measures of 
opportunities, and among a wide range of models and approaches, provides strong evidence 
for the five key takeaways from our analysis. 

1. On average, students who attended the network schools in the study reported greater 
opportunities to engage in deeper learning than did similar students who attended 
non-network schools. Analyses of student survey data confirmed the first core assumption 
of the deeper learning initiative: that network schools provided students with greater 
opportunities to engage in deeper learning. These findings were consistent across all 
measures of opportunities aligned with the deeper learning competencies. In addition, we 
observed positive effects of network participation for at least one opportunity measure in 
all individual pairs of schools; and we observed no negative effects. These consistent 
positive findings are notable, particularly given the wide range of approaches and practices 
to promote deeper learning that were implemented in the participating schools. While we 
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observed some commonalities in the strategies, structures, and cultures schools used 
to promote deeper learning, each network had a unique approach that emphasized the 
development of deeper learning competencies in different ways. The common positive 
influence these approaches had on the opportunities experienced by students suggests 
that deeper learning opportunities can be provided through many different models and 
approaches. By demonstrating that these schools are achieving what they set out to 
achieve in terms of opportunities for students, this finding lays the groundwork for 
additional research into the effect of network participation on student outcomes. 

2. These differences in opportunities were observed among schools serving diverse 
student populations, including substantial subgroups of traditionally underserved 
students. The network schools did not base admissions on students’ prior academic 
achievement, and therefore served students of all ability levels. In addition, only schools 
with at least 30 percent eligibility for free or reduced price lunch participated in the study. 

In many cases, the participating schools had a much higher percentage of students who 
were eligible for free or reduced price lunch (e.g., in two schools, 100 percent of students 
were eligible), and a subset of schools included substantial populations of English language 
learners. The consistent positive findings from this proof-of-concept analysis therefore 
demonstrate that these types of opportunities can be and were provided to a diverse 
group of students, including traditionally underserved student populations. 

3. The effects of attending a participating network school on deeper learning opportunities 
were similarly positive for subgroups of students including initially high- and low-achievers 
and students who did or did not qualify for free or reduced price lunch. To further explore 
the consistency of opportunities across subgroups of students, including high-need student 
populations, we examined whether students with incoming high or low achievement 
experienced significantly different effects of network participation. We found that they did 

not. This finding demonstrates that the participating schools were successful in providing 
deeper learning opportunities to the full range of students, not just those who had 
experienced greater academic success in the past or were enrolled in more advanced 
courses. In addition, we also found no difference in effect for students who did or did not 
qualify for free or reduced price lunch. Providing opportunities equitably to all students is a 
core goal of the Hewlett Foundation’s deeper learning initiative. This finding again suggests 
that the schools are accomplishing what they set out to achieve in terms of providing 
deeper learning opportunities to all students. 

4. Teachers’ most challenging assignments collected from the network schools exhibited 
greater opportunities for independent learning in mathematics and real-world connections 
in ELA than the challenging assignments collected from the non-network schools, but 
were not significantly different on other opportunity measures (including complex 
problem solving, communication, and conceptual understanding in mathematics). 
Findings also indicated that in both network and non-network schools, there was room 
for improvement in the opportunities for deeper learning provided to students through 
even the most challenging assignments. 
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5. The opportunities for deeper learning experienced by individual students, regardless of 
whether they attended a network school, were associated with those students’ deeper 
learning outcomes. An analysis of the relationship between opportunities and outcomes 
demonstrated an association between opportunities for deeper learning and interpersonal 
and intrapersonal competencies, as well as the ability to solve complex problems on a 
standardized assessment. We found a significant relationship between opportunities for 
deeper learning and outcomes of interest for individual students across all participating 
schools, regardless of network affiliation. Having confirmed this relationship, any differences 
in outcomes between network and non-network schools can more clearly be attributed to 
schools’ focus on providing opportunities for deeper learning. 

The differences we found in the student survey responses indicate that there was a meaningful 
difference in students’ experienced learning opportunities between those attending network and 
non-network schools, including for subgroups of traditionally underserved students. These findings 
demonstrate that students’ learning experiences in the network schools were indeed different 
than what comparable students experienced in other schools, and that students engaged in more 
opportunities for deeper learning across their core content classes. Since the sample of schools 
included a wide range of approaches to promoting deeper learning (see our first report), these 
results also suggest that these opportunities can be provided in many different ways. Nonetheless, 
our survey measures did not examine the quality of the opportunities nor the frequency with which 
they were experienced within classes. In addition, teacher assignment data suggested that the 
opportunities provided at the network schools through the most challenging assignments could 
be further improved. Therefore, while these results suggest that the network schools have 
made significant changes in students’ learning experiences, in comparison to more traditional 
approaches, they also suggest that more investigation could be done to better understand and 
define these experiences, and that continued work could be done by network schools to offer 
greater quality opportunities. 

In sum, we believe that these findings confirm one of the primary assumptions outlined in the 
abbreviated theory of action described in the introduction to this report - that explicit strategies, 
structures, and school culture designed to support deeper learning would provide students with 
greater opportunities to engage in deeper learning. The findings in this report also help to confirm 
another assumption of this theory of action - that a relationship exists between individual students' 
experienced opportunities for deeper learning and student outcomes. These findings lay the 
foundation for our analysis of deeper learning outcomes, which is summarized in our third report, 
Evidence of Deeper Learning Outcomes (Zeiser, Taylor, Rickies, Garet, & Segeritz, 2014). In that 
report, we discuss the differences between network and non-network student outcomes, including 
interpersonal and intrapersonal outcomes, performance on the OECD PISA-Based Test for Schools, 
state test scores, graduation rates, and postsecondary enrollment. Taken together, this series 
of reports provides an overall picture of deeper learning network approaches, as well as the 
opportunities and outcomes associated with these approaches. 
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